Skip to content

Conversation

@andrew-anyscale
Copy link
Contributor

@andrew-anyscale andrew-anyscale commented Jan 7, 2026

This adds wanda-based builds for anyscale images used in release tests.

Changes:

  • Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images
  • Add ray-llm-anyscale-cuda.wanda.yaml for LLM images
  • Add ray-ml-anyscale-cuda.wanda.yaml for ML images
  • Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane
  • Update release/build.rayci.yml with build and push steps
  • Update gcloud_docker_login.sh to v550.0.0 with arch detection

Topic: anyscale-image
Relative: ray-image
Labels: draft
Signed-off-by: andrew [email protected]

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

Reviews in this chain:
#60197 [ci] Add RayImagePushContext for publishing wanda Ray images
 └#59936 feat(ci): wanda ray image builds, uploads to Dockerhub
  └#59937 Add wanda anyscale image builds for release tests

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

# head base diff date summary
0 f963b425 f104bf59 diff Jan 7 7:57 AM 8 files changed, 553 insertions(+), 24 deletions(-)
1 07109714 288cadb6 diff Jan 7 7:58 AM 0 files changed
2 4cdb2dd9 46cc8625 diff Jan 7 8:38 AM 2 files changed, 238 insertions(+), 5 deletions(-)
3 f65669a5 70df7561 rebase Jan 7 11:29 AM 0 files changed
4 11418977 ae32d45a rebase Jan 7 12:21 PM 0 files changed
5 af3b600e 04cfd26a rebase Jan 7 17:29 PM 0 files changed
6 6bbeb9e4 55d1473f rebase Jan 7 17:31 PM 0 files changed
7 1f049e81 84d74797 diff Jan 8 8:27 AM 2 files changed, 12 insertions(+), 12 deletions(-)
8 d43ccd7f c275fe24 rebase Jan 8 8:32 AM 0 files changed
9 8980d9ce 7c78fa2d rebase Jan 8 12:01 PM 0 files changed
10 4a143b7a 29cf5abd rebase Jan 8 12:51 PM 0 files changed
11 9243672a d566c8a8 rebase Jan 8 13:16 PM 0 files changed
12 8c684c8e 91900297 rebase Jan 8 13:33 PM 0 files changed
13 826bac0f 73822f53 rebase Jan 8 14:30 PM 0 files changed
14 ebdca80e 94805e95 rebase Jan 9 8:45 AM 0 files changed
15 35249bf7 fc79d67d rebase Jan 9 10:01 AM 0 files changed
16 0e1bd8b1 db94f935 rebase Jan 9 10:59 AM 0 files changed
17 5de31a90 05c06707 rebase Jan 9 12:36 PM 0 files changed
18 a96c770e a6718508 rebase Jan 9 13:32 PM 0 files changed
19 47087a0b 51fe33a5 rebase Jan 13 8:55 AM 0 files changed
20 5b7b8097 ed74c8c7 rebase Jan 13 10:55 AM 0 files changed
21 827515b8 6242ab25 rebase Jan 13 15:50 PM 0 files changed
22 c290dfca a941fd54 rebase Jan 13 15:50 PM 0 files changed
23 da45cd89 9db9513a rebase Jan 13 17:03 PM 0 files changed
24 79224a60 6d4a0beb rebase Jan 13 17:09 PM 0 files changed
25 3123fe23 dd5133b8 rebase Jan 13 17:09 PM 0 files changed
26 a57ba933 ae00964a rebase Jan 13 17:12 PM 0 files changed
27 abd456d2 021b2e13 rebase Jan 13 17:14 PM 0 files changed
28 2dc9e7bb 52b7b213 rebase Jan 13 17:33 PM 0 files changed
29 5e859eea 8cfa977c rebase Jan 13 21:52 PM 0 files changed
30 dc8bd484 924d46f1 rebase Jan 15 13:10 PM 0 files changed
31 fc711fb0 275827a2 rebase Jan 15 13:13 PM 0 files changed
32 acdfe07a 04d68065 rebase Jan 15 13:14 PM 0 files changed
33 50ffff8a 6e985e93 rebase Jan 15 13:40 PM 0 files changed
34 b07e1394 17d793f4 rebase Jan 15 13:47 PM 0 files changed
35 77436f88 716cc651 rebase Jan 15 16:04 PM 0 files changed
36 71b04f1e 976be9fb rebase Jan 15 16:48 PM 0 files changed
37 ebae2c55 f4e1306c diff Jan 15 17:41 PM 5 files changed, 12 insertions(+), 8 deletions(-)
38 96eb30f9 73da8387 rebase Jan 15 18:02 PM 0 files changed

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Anyscale image build process to use Wanda for building and a new Python script for pushing images to multiple registries. The changes include new Wanda configuration files, the push_anyscale_image.py script, and updates to the Buildkite pipeline. My review identifies a few critical issues that will break the build, including a missing file in a Bazel target and an incomplete matrix configuration in the pipeline. I've also provided several suggestions to improve maintainability by reducing code duplication and making parts of the new script less brittle.

Comment on lines 321 to 355
py_binary(
name = "push_ray_image",
srcs = ["push_ray_image.py"],
exec_compatible_with = ["//bazel:py3"],
deps = [
":crane_lib",
"//ci/ray_ci:ray_ci_lib",
ci_require("click"),
],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This py_binary target for push_ray_image references a source file push_ray_image.py that is not included in this pull request. This will cause the Bazel build to fail. Please either add the missing file or remove this target if it's not intended to be part of this change.

Comment on lines +171 to +174
- bash release/gcloud_docker_login.sh release/aws2gce_iam.json
- bash release/azure_docker_login.sh
- az acr login --name rayreleasetest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These authentication commands are duplicated across the anyscalebuild, anyscalellmbuild, and anyscalemlbuild push jobs. To improve maintainability and reduce redundancy, consider using YAML anchors and aliases to define these commands once and reuse them in each job. This will make future updates to the authentication process much easier.


# GPU_PLATFORM is the default GPU platform that gets aliased as "gpu"
# This must match the definition in ci/ray_ci/docker_container.py
GPU_PLATFORM = "cu12.1.1-cudnn8"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The GPU_PLATFORM is hardcoded here. The comment mentions it must match a definition in another file, which makes this brittle. If the default GPU platform changes, this script will require a manual update, which can be easily missed. Consider sourcing this value from a shared configuration file or passing it as a command-line argument to make the script more robust and maintainable.

Comment on lines +69 to +75
def _format_platform_tag(platform: str) -> str:
"""Format platform as -cpu or shortened CUDA version."""
if platform == "cpu":
return "-cpu"
# cu12.3.2-cudnn9 -> -cu123
versions = platform.split(".")
return f"-{versions[0]}{versions[1]}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for parsing the CUDA version string using platform.split('.') is fragile. It assumes a specific format with dots as separators and might fail with an IndexError if the platform string format changes (e.g., cu12-cudnn8). Using a regular expression would be more robust for extracting the version parts. For example: re.match(r"cu(\\d+)\\.(\\d+)", platform).

Comment on lines +78 to +127
def _get_image_tags(python_version: str, platform: str) -> List[str]:
"""
Generate image tags matching the original docker_container.py format.
Returns multiple tags for the image (canonical + aliases).
For GPU_PLATFORM, also generates -gpu alias tags to match release test expectations.
"""
branch = os.environ.get("BUILDKITE_BRANCH", "")
commit = os.environ.get("BUILDKITE_COMMIT", "")[:6]
rayci_build_id = os.environ.get("RAYCI_BUILD_ID", "")

py_tag = _format_python_version_tag(python_version)
platform_tag = _format_platform_tag(platform)

# For GPU_PLATFORM, also create -gpu alias (release tests use type: gpu)
platform_tags = [platform_tag]
if platform == GPU_PLATFORM:
platform_tags.append("-gpu")

tags = []

if branch == "master":
# On master, use sha and build_id as tags
for ptag in platform_tags:
tags.append(f"{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")
elif branch.startswith("releases/"):
# On release branches, use release name
release_name = branch[len("releases/") :]
for ptag in platform_tags:
tags.append(f"{release_name}.{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")
else:
# For other branches (PRs, etc.)
pr = os.environ.get("BUILDKITE_PULL_REQUEST", "false")
if pr != "false":
for ptag in platform_tags:
tags.append(f"pr-{pr}.{commit}{py_tag}{ptag}")
else:
for ptag in platform_tags:
tags.append(f"{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")

return tags
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This function has a lot of repeated code, especially the block for appending rayci_build_id tags, which is identical in all if/elif/else branches. This can be refactored to be more DRY (Don't Repeat Yourself), which improves readability and maintainability. You could determine the tag prefix first, then generate the tags in a single block of code.

def _get_image_tags(python_version: str, platform: str) -> List[str]:
    """
    Generate image tags matching the original docker_container.py format.

    Returns multiple tags for the image (canonical + aliases).
    For GPU_PLATFORM, also generates -gpu alias tags to match release test expectations.
    """
    branch = os.environ.get("BUILDKITE_BRANCH", "")
    commit = os.environ.get("BUILDKITE_COMMIT", "")[:6]
    rayci_build_id = os.environ.get("RAYCI_BUILD_ID", "")

    py_tag = _format_python_version_tag(python_version)
    platform_tag = _format_platform_tag(platform)

    # For GPU_PLATFORM, also create -gpu alias (release tests use type: gpu)
    platform_tags = [platform_tag]
    if platform == GPU_PLATFORM:
        platform_tags.append("-gpu")

    tags = []
    if branch == "master":
        prefix = commit
    elif branch.startswith("releases/"):
        release_name = branch[len("releases/") :]
        prefix = f"{release_name}.{commit}"
    else:
        pr = os.environ.get("BUILDKITE_PULL_REQUEST", "false")
        if pr != "false":
            prefix = f"pr-{pr}.{commit}"
        else:
            prefix = commit

    for ptag in platform_tags:
        tags.append(f"{prefix}{py_tag}{ptag}")

    if rayci_build_id:
        for ptag in platform_tags:
            tags.append(f"{rayci_build_id}{py_tag}{ptag}")

    return tags

@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 0710971 to 4cdb2dd Compare January 7, 2026 16:38
@andrew-anyscale andrew-anyscale changed the base branch from andrew/revup/master/ray-wheel to andrew/revup/master/ray-image January 7, 2026 16:38
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 46cc862 to 70df756 Compare January 7, 2026 19:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 4cdb2dd to f65669a Compare January 7, 2026 19:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 70df756 to ae32d45 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from f65669a to 1141897 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from ae32d45 to 04cfd26 Compare January 8, 2026 01:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from af3b600 to 6bbeb9e Compare January 8, 2026 01:31
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 04cfd26 to 55d1473 Compare January 8, 2026 01:31
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 6bbeb9e to 1f049e8 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 55d1473 to 84d7479 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 84d7479 to c275fe2 Compare January 8, 2026 16:32
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from d43ccd7 to 8980d9c Compare January 8, 2026 20:01
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 7c78fa2 to 29cf5ab Compare January 8, 2026 20:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 8980d9c to 4a143b7 Compare January 8, 2026 20:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 29cf5ab to d566c8a Compare January 8, 2026 21:16
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 4a143b7 to 9243672 Compare January 8, 2026 21:16
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from d566c8a to 9190029 Compare January 8, 2026 21:33
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 79224a6 to 3123fe2 Compare January 14, 2026 01:09
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from dd5133b to ae00964 Compare January 14, 2026 01:12
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from a57ba93 to abd456d Compare January 14, 2026 01:14
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from ae00964 to 021b2e1 Compare January 14, 2026 01:14
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from abd456d to 2dc9e7b Compare January 14, 2026 01:33
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 021b2e1 to 52b7b21 Compare January 14, 2026 01:33
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 2dc9e7b to 5e859ee Compare January 14, 2026 05:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 52b7b21 to 8cfa977 Compare January 14, 2026 05:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 5e859ee to dc8bd48 Compare January 15, 2026 21:10
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 8cfa977 to 924d46f Compare January 15, 2026 21:10
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from dc8bd48 to fc711fb Compare January 15, 2026 21:13
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 924d46f to 275827a Compare January 15, 2026 21:13
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from fc711fb to acdfe07 Compare January 15, 2026 21:14
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 04d6806 to 6e985e9 Compare January 15, 2026 21:40
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from acdfe07 to 50ffff8 Compare January 15, 2026 21:40
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 6e985e9 to 17d793f Compare January 15, 2026 21:47
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 50ffff8 to b07e139 Compare January 15, 2026 21:47
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 17d793f to 716cc65 Compare January 16, 2026 00:04
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from b07e139 to 77436f8 Compare January 16, 2026 00:04
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 716cc65 to 976be9f Compare January 16, 2026 00:48
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 77436f8 to 71b04f1 Compare January 16, 2026 00:48
This adds wanda-based builds for anyscale images used in release tests.

Changes:
- Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images
- Add ray-llm-anyscale-cuda.wanda.yaml for LLM images
- Add ray-ml-anyscale-cuda.wanda.yaml for ML images
- Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane
- Update release/build.rayci.yml with build and push steps
- Update gcloud_docker_login.sh to v550.0.0 with arch detection

Topic: anyscale-image
Relative: ray-image
Labels: draft
Signed-off-by: andrew <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants